On the Power of Semantic Partitioning of Web Documents

A growing number of Web sites are maintained by content management software and thus a large number of Web pages are machine-generated via templates. Normally in such Web pages there is implicitly a fixed “schema” and what changes is the content. Informally a schema for a Web page represents concepts and relationships among them in a hierarchical fashion. For example, Figure 1 is a screen shot of the New York Times front page (see http://www.nytimes.com). Observe that this page includes: (i) a taxonomy of items such as “NEWS” (consisting of hyperlinks labeled with “International”, “National”, ...), “OPINION” (consisting of hyperlinks “Editorial/Op-Ed”, ...), etc.; (ii) several headlines of news articles where each article begins with a hyperlink labeled with the news headline (e.g., “Bush tells Nation ...”) followed by the author of the article (e.g., “By Richard W. Stevenson ...”), followed by a time-stamp and a text summary of the article (e.g., “President Bush portrayed ...”). The schema for this fragment of the New York Times front page therefore includes the taxonomy (which does not change) and the template for the news article. We should point out that the schema will also include several additional elements pertaining to other content appearing in the page.